The present invention relates to a method of data multiplexing utilized in multiplexing digital data including pictures, audio and characters, and transmitting or recording the resultant multiplex data, a method of decoding the multiplexed data, a decoder using the decoding method, a multiplexed data repeater disposed between a multiplexer and the decoder, and a recording medium in which the methods are recorded.
An international standard called xe2x80x9cMPEG 2xe2x80x9d is a method of multiplexing digital data such as pictures, audio and characters. (Reference Documents: ISO/IEC JTC1/SC29/WG11 N801, xe2x80x9cISO/IEC 13818-1 International Standard: Information Technologyxe2x80x94Generic Coding of Moving Pictures and Associated Audio Information: Systemsxe2x80x9d, 1994. 11)
FIG. 13 is a block diagram of a multiplexing system that multiplexes pictures and audio data coded in the MPEG 2 format. The multiplexing system surrounded by a broken line comprises a picture data buffer 1305, a audio data buffer 1306, a packetizer 1307, a padding packet buffer 1309, a multiplexer 1311, and a timing generator 1314. A picture encoder 1301 and an audio encoder 1303 are coupled to this multiplexing system.
FIG. 14 is a flowchart depicting an operation of a selector 1311. FIG. 15 is a schematic diagram illustrating a data structure of MPEG 2. FIG. 16 illustrates a method of multiplexing data in MPEG 2 format. A conventional multiplexing method in MPEG 2 format is described hereinafter with reference to FIGS. 13, 14, 15 and 16.
FIG. 15 is a schematic diagram illustrating a data structure of MPEG 2. A picture or an audio data shown in FIG. 15(a) is cut out in a given size, then a header is attached to the taken-out picture or audio data as shown in FIG. 15(b) before being packetized into a PES packet. As shown in FIG. 15(c), the header includes an identifier for identifying a kind of data, a decoding time at which a frame of picture or an audio data should be decoded as well as a reproduction time at which the frame of the picture or the audio data should be reproduced. One picture frame comprises one picture, and an audio frame comprises e.g. 1152 audio samples. Further, as shown in FIG. 15(d) and FIG. 15(e), another header called a pack header is included in the packet. The pack header includes a sample value of a reference time that is used for reproducing the reference time that is produced in the decoder.
An operation of the multiplexing system shown in FIG. 13 is described hereinafter. The picture encoder 1301 encodes (compression) one picture frame and outputs a picture data 1302. An audio encoder 1303 encodes one frame of audio data and outputs an audio data 1304. The picture data and the audio data supplied to the multiplexing system are stored in the respective buffers 1305 and 1306. The timing generator 1314 generates a timing with which a packet should be supplied, and inputs the packet generating signal into the multiplexer 1311. The multiplexer operates following the flowchart shown in FIG. 14. When the packet generating signal 1315 is supplied in step 1401, a packet is selected for outputting process. In this process, it is assumed that a decoder is employed in the later process, and the packet is selected so that the buffer in the assumed decoder for pictures or audio data could avoid being overflowed or underflowed. A buffer capacity for pictures or audio data is standardized by MPEG 2, e.g. 1.8 M bit for pictures, 4096 bytes for audio data.
First, in step 1402, when a vacant capacity for one packet is found in the audio data buffer of the assumed decoder, an audio-packet-generating-signal 1313 is supplied to the packetizer 1307 in step 1403. The packetizer receives audio data for one packet from the buffer 1305, and outputs an audio packet 1308. When a vacant capacity for one packet is not found in the audio data buffer, while it is found in the picture data buffer of the assumed decoder in step 1404, a picture packet generating signal 1313 is supplied to the packetizer in step 1405. The packetizer receives a packet data for one packet from the buffer 1306 and outputs a picture packet 1308. In step 1407, when the multiplexer receives an audio packet or a picture packet, a reference time is attached to the received packet. When neither the audio data buffer nor picture data buffer has a vacant capacity for one packet, a padding packet 1310 is supplied to the multiplexer from a padding packet buffer 1309 in step 1406. In step 1408, the multiplexer outputs a received packet 1312.
FIG. 16 illustrates transitional occupied capacities of the picture data and audio data buffers, which are used in the multiplexer 1311 shown in the flowchart of FIG. 14. FIG. 16(b) depicts the transitional occupied capacity of audio data buffer. The X-axis indicates time xe2x80x9ctxe2x80x9d that is a reference time reproduced from the reference time sample value shown in FIG. 15. A data for one audio frame is decoded at the time of xe2x80x9cta1xe2x80x9d, xe2x80x9cta2xe2x80x9d, . . . respectively, and they are deleted from the buffer. The time, xe2x80x9cta1xe2x80x9d, xe2x80x9cta2xe2x80x9d, . . . , are the decoding time included in the packet header shown in FIG. 15(c). The Y-axis indicates a packet capacity occupied by data, and a broken line indicates a buffer size xe2x80x9cSaxe2x80x9d. FIG. 16(c) depicts the transitional occupied capacity of picture data buffer. A data for one picture frame is decoded at the time of xe2x80x9ctv1xe2x80x9d, xe2x80x9ctv2xe2x80x9d, . . . respectively, and they are deleted from the buffer. The Y-axis indicates a packet capacity occupied by data, and a broken line indicates a buffer size xe2x80x9cSvxe2x80x9d. FIG. 16(a) depicts a packet stream 1312 that is tapped off from the selector 1311. Until the time xe2x80x9cta1xe2x80x9d, a vacant capacity for one packet is not available in the audio data buffer, while it is available in the picture data buffer. A picture packet is thus multiplexed. Because an audio frame is decoded at xe2x80x9cta1xe2x80x9d, an audio packet is multiplexed. At just before xe2x80x9ctv4xe2x80x9d, because neither of the audio packet nor picture packet has a vacant capacity for one packet, a padding packet is multiplexed.
FIG. 17 illustrates a conventional decoder for multiplexed data coded in MPEG 2 format. The decoder comprises the following elements: separator 1702, picture data buffer 1704, audio data buffer 1706, reference time reproduction part 1708, picture data decoder 1710, audio data decoder 1712.
The separator 1702 receives and separates a multiplexed data 1701 into a picture packet data 1703 and an audio packet data 1705 referring to the identifier attached to the packet header. The resultant separated data 1701 and 1703 are stored in the respective buffers 1704 and 1706. At the same time, a sample value 1707 of the reference time is extracted and supplied to the reference time reproduction part 1708, where a reference clock of the decoder is produced. The reproduction part 1708 outputs a reference time signal 1709 to the picture data decoder 1710 as well as to the audio data decoder 1712. The picture data decoder 1710 compares the decoding time included in the picture-packet-header that is stored in the buffer 1704 with the reference time signal. When the decoding time comes, the picture data is decoded and a picture frame 1711 is reproduced. In the same manner, the audio decoder 1712 compares the decoding time included in the-audio-packet-header that is stored in the buffer 1706 with the reference time signal. When the decoding time comes, the audio data is decoded and an audio frame 1713 is reproduced.
A method of object coding has drawn attention as a new coding method. This method divides a picture into objects such as a plurality of pictures physically including background and foreground pictures as well as audio accompanying the respective pictures. The conventional coding method of picture and audio has handled a picture and audio data thereof hitherto as a whole picture and audio accompanying the picture. An advantage of this object coding method is to lower a transmission rate by the following methods: (a) replace an original background with another background, (b) delete or add an object arbitrary, (c) select a particular object to transmit.
The multiplexing method that handles a picture in a frame unit coded in MPEG 2 format as described in the prior art has encountered various problems. For instance, in a case where respective object data are multiplexed and supplied to a multiplexed data decoder, it is difficult to manage the data buffers, because the respective object data are to be stored in the data buffers after the multiplexed data are separated and picture data as well as audio data are decoded.
It is necessary to assign the data buffer to the respective object data in the decoder; however, no management method is available. A total buffer capacity required by the respective objects sometimes exceeds the data buffer capacity of the decoder. When it happens, all or some of the objects cannot be correctly decoded. An object including an important message can be thus missed.
Further there are a few problems. On a transmission line between the multiplexer and the decoder, no method is ready to select some objects when all the objects cannot be multiplexed. When the transmission line between the multiplexer and the decoder is changed at a relay point in between, all the objects arrived there cannot transmitted further. The decoder cannot thus produce a correct decoded result.
A data multiplexing method of the first invention addressing the problems mentioned above is described hereinafter. This multiplexing method comprises the following steps:
(a) receiving and multiplexing N pieces of digital data such as pictures, audio or characters (N=natural number);
(b) preparing a buffer capacity necessary for decoding N pieces of digital data;
(c) comparing the prepared buffer capacity with a given capacity;
(d) selecting M pieces of digital data out of N pieces of digital data so that a total decoded buffer capacity thereof can be not more than the given capacity (M=natural number, and 1xe2x89xa6Mxe2x89xa6N).
(e) multiplexing the M pieces of digital data; and
(f) outputting the multiplexed digital data as one multiplexed data.
This method can effect the following advantage. When coded data is transmitted between systems of which decoders have different decoding buffer capacities, decoding buffer capacities of the receiver sides should be informed before coded data are transmitted. Then a given capacity set in the transmitter side is replaced with this informed capacity responsive to the receiver. The object data thus can be decoded correctly and free from missing an important object.
A data multiplexing method of the second invention is described hereinafter. This multiplexing method comprises the following steps.
(a) receiving and multiplexing N pieces of object data such as pictures, audio or characters (N=natural number);
(b) supplying a transmission rate necessary for outputting each one of the respective N pieces object data;
(c) finding a total of the transmission rates necessary for each object data, and referring it as a first transmission rate
(d) supplying a transmission rate of the transmission line where an output from the multiplexer is transmitted, and referring it as a second transmission rate;
(e) selecting M pieces of object data out of N pieces so that the first transmission rate (totaled transmission rates of respective M pieces objects) can be not more than the second transmission rate (transmission rate of the transmission line); and
(f) multiplexing the selected M pieces objects data.
A data multiplexing method of the third invention comprises the following steps.
(a) receiving N pieces of object data including pictures, audio or characters, and multiplexing them;
(b) producing a table data where at least one of the following data are described:
(b-1) a number of digital data to be multiplexed;
(b-2) decoding buffer capacities of each digital data;
(b-3) coding rates of each digital data;
(b-4) priority order of each digital data; and
(b-5) identifiers of each digital data, which identify the data as a picture, audio, characters or another data.
(c) multiplexing the table data to the multiplexed data so that decoding buffer of the decoder can be assigned to each object with ease.
A decoding multiplexed data of the fourth invention is described hereinafter. The decoding method comprises the following steps.
(a) receiving N pieces of object data including pictures, audio or characters in a multiplexed form with a predetermined unit such as an object unit (N=natural number);
(b) checking a decoding buffer capacity, and referring it as a first decoding buffer capacity;
(c) obtaining respective decoding buffer capacities necessary for decoding N pieces of digital data;
(d) finding a total decoding buffer capacity of the N pieces digital data, and referring it as a second decoding buffer capacity;
(e) checking whether the first decoding buffer capacity exceeds the second one; and
(f) when the first capacity exceeds the second one, selecting M pieces of digital data from the multiplexed data and outputting them (1xe2x89xa6Mxe2x89xa6N).
Through these steps, the data that the transmitter desires to send can be exactly decoded and reproduced. The same process as mentioned above can be applicable to the transmission rate with the same effect.
A multiplexed data repeater of the fifth invention solves the problem, i.e. a transmission rate is changed between the multiplexer and the decoder, by the following methods.
(a) receiving N pieces of digital data including pictures, audio or characters in a multiplexed form (N=natural number);
(b) selecting part of the data out of the multiplexed data; and
(c) outputting the selected data changing the transmission rate, thereby transmitting the data exactly even if a transmission rate between the multiplexer and the decoder is changed.
A multiplexed data decoder of the sixth invention performs the following functions:
(a) receiving N pieces of object data including pictures, audio or characters in a multiplexed form (N=natural number);
(b) assigning the decoding buffer of the decoder to respective objects;
(c) dividing the decoding buffer into N pieces of sub-decoding buffers; and
(d) storing the separated each object data in the respective sub-decoding buffers.
A recording medium of the seventh invention contains the methods of data multiplexing and a method of multiplexed data decoding mentioned above. When this medium is incorporated into a computer system, the data multiplexing system, multiplexed data repeater and multiplexed data decoder of the present invention can be effected with ease.
As such, the data multiplexing method, multiplexed data decoding method, and the devices employing these methods can effect an exact decoding of object data and a transmission free from missing an important message even between the systems having different decoding buffer capacities in their decoders. This advantage is executed in the following manner. Before transmitting coded data, the decoding buffer capacity on receiver sides should be informed. A given buffer capacity can be changed responsive to the capacity of the receiver. On the receiver side, on the other hand, if the decoding buffer capacity is insufficient for decoding the multiplexed data just received, the priority order of data units, e.g. an object unit, on the transmitter side should be obtained. Then data units are selected according to the priority order before they are decoded so that an important message can be exactly decoded.
Changing a given decoding buffer capacity in the system allows a user to transmit coded object data free from concerning the data capacity, and the data are decoded without missing an important message.